The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @zucchini-nlp do you know what the popular VLMs using reasoning or tool calls are right now? I'd like to add support + testing in the processor too.

From the ones we have already in library, Ovis2 chat templates has some parts for tool usage. Other than that, haven't seen explicit reasoning/tools in template

Thank you! I'll take a look at Ovis2, if not I can worry about adding Processor schemas when we actually need them

Made a quick demo so reviewers can try this out. Just switch to the chat_schemas branch and then:

from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's 6 * 8?"
messages = [
    {"role": "user", "content": prompt}
]

out = pipe(messages, max_new_tokens=512)
print(out[0]["generated_text"][-1])

If it works, you should see the output correctly split into thinking and content blocks!

And here's a quick demo for tool calling:

from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

def get_current_weather(location: str):
    """
    Gets the weather at a given location

    Args:
    location: The location to get the weather for
    """
    return 20.

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's the weather like in Paris?"
messages = [
    {"role": "user", "content": prompt}
]

out = pipe(messages, tools=[get_current_weather], max_new_tokens=512)
print(out[0]["generated_text"][-1])

The tool call should be correctly parsed as a key in the response dict.

Hey, I've recently been working on the tool call handling in vLLM and came across this. Lots of really cool stuff going on here!

A couple questions I have:

• Would it be possible to support using a library instead of a regex for parsing the output from the model? The Harmony library from OpenAI is pretty complicated already and replicating the logic in a regex would be pretty tough. For example:
  • Header parsing has some logic where it splits on whitespace to get the content_type of a tool call
  • There are two valid places for recipient to be in the headers
• Would it be possible to implement this at the token level instead of after decoding? Doing it at the token level would remove the risk of using text versions of special tokens to break parsing.

• Would it be possible to implement this at the token level instead of after decoding? Doing it at the token level would remove the risk of using text versions of special tokens to break parsing.

To this point, the Harmony library expects to deal directly with token ids, right? So if we allow integration with model provider libraries for this parsing, we may have to support token ids.

I think that also aligns very well with the goal of this PR. The difference is that this API focuses more on high-level semantics, whereas Structural Tag emphasizes structure at the raw text level. I believe there’s a lot of space for collaboration here, and I’d love to explore this further.

I also would like to see a push for xgrammar upstream as well here.

@Ubospica yes, Structural Tag looks very similar to this! Constrained generation and output parsing obviously have a lot of overlap, since they both define an output schema of some kind. Do you know how output parsing was intended to be implemented for Structural Tag?

Actually, a better question: If we want to align with XGrammar, should we try to extend the Structural Tag spec to allow output parsing, or should we have an output schema for parsing that's separate from the Structural Tag?

I thought about it for a while and here's what I have:

The main thing response parsing needs that structural tag doesn't have is a way to map segments of the output to paths in the parsed message dict. For example, it is very common that LLMs render tool calls or tool definitions as JSON schema, but not in the standard OpenAI format our API expects for tool calls. For example, they may rename "arguments" to "parameters", or they may leave out the "function" key, etc.

The goal of chat parsing is that the model should return a message dictionary that is ready to be appended to the chat without any further parsing from the user, which means it must be in the standard API format. This means we need a way to map "segments" of the structural tag schema to segments of the output dict. If the model emits:

<tool_call>
{
   "name": "get_current_weather",
   "parameters": {
       "location": "Paris"
    }
}
</tool_call>

The output we want from parsing is:

{
    "role": "assistant",
    "tool_calls": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "arguments": {
                    "location": "Paris"
                }
           }
       ]
}

It's easy to express the model output above as a structural tag schema, but it's trickier to do the mapping from there to the output format we want. It may even be strictly impossible in some cases - some of the structural tag keys like regex and particularly grammar don't seem like they have a good way to separate internal segments. With regex we could have named capturing groups, maybe, but with EBNF grammars it won't work, without some nonstandard additions to the EBNF spec. Even if we could make it work, it seems like it would clutter the structural tag spec, which right now is clean and usable.

Maybe it makes more sense to keep response parsing separate, but I can upstream it or something like it to XGrammar as a separate feature, alongside structural tags for constrained generation?

For information, multiple tool calls isn't supported yet:

from transformers import pipeline

model_name = "Rocketknight1/qwen-response-test"

def get_current_weather(location: str):
    """
    Gets the weather at a given location

    Args:
    location: The location to get the weather for
    """
    return 20.

pipe = pipeline("text-generation", model_name, dtype="auto", device_map="auto")

prompt = "Hey, what's the weather like in Paris and London?"
messages = [
    {"role": "user", "content": prompt}
]

tools = [get_current_weather]
out = pipe(messages, tools=tools, max_new_tokens=512)
print(out[0]["generated_text"][-1])

Traceback (most recent call last):
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 114, in recursive_parse
    parsed_json = json.loads(node_content)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/json/decoder.py", line 341, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 3 column 1 (char 69)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/fsx/qgallouedec/trl/parsing.py", line 22, in <module>
    out = pipe(messages, tools=tools, max_new_tokens=512)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 325, in __call__
    return super().__call__(Chat(text_inputs), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1289, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/base.py", line 1297, in run_single
    outputs = self.postprocess(model_outputs, **postprocess_params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/pipelines/text_generation.py", line 530, in postprocess
    assistant_message = self.tokenizer.parse_response(all_text)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/tokenization_utils_base.py", line 1803, in parse_response
    return recursive_parse(response, schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 173, in recursive_parse
    parsed_schema[key] = recursive_parse(node_content[key], child_node)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/utils/chat_parsing_utils.py", line 116, in recursive_parse
    raise ValueError(
ValueError: Node has JSON parser but could not parse its contents as JSON: 
{"name": "get_current_weather", "arguments": {"location": "Paris"}}
</tool_call>

<tool_call>
{"name": "get_current_weather", "arguments": {"location": "London"}}

Error: Extra data: line 3 column 1 (char 69)

I thought about it for a while and here's what I have:

The main thing response parsing needs that structural tag doesn't have is a way to map segments of the output to paths in the parsed message dict. For example, it is very common that LLMs render tool calls or tool definitions as JSON schema, but not in the standard OpenAI format our API expects for tool calls. For example, they may rename "arguments" to "parameters", or they may leave out the "function" key, etc.

The goal of chat parsing is that the model should return a message dictionary that is ready to be appended to the chat without any further parsing from the user, which means it must be in the standard API format. This means we need a way to map "segments" of the structural tag schema to segments of the output dict. If the model emits:

<tool_call>
{
   "name": "get_current_weather",
   "parameters": {
       "location": "Paris"
    }
}
</tool_call>

The output we want from parsing is:

{
    "role": "assistant",
    "tool_calls": [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "arguments": {
                    "location": "Paris"
                }
           }
       ]
}

It's easy to express the model output above as a structural tag schema, but it's trickier to do the mapping from there to the output format we want. It may even be strictly impossible in some cases - some of the structural tag keys like regex and particularly grammar don't seem like they have a good way to separate internal segments. With regex we could have named capturing groups, maybe, but with EBNF grammars it won't work, without some nonstandard additions to the EBNF spec. Even if we could make it work, it seems like it would clutter the structural tag spec, which right now is clean and usable.

Maybe it makes more sense to keep response parsing separate, but I can upstream it or something like it to XGrammar as a separate feature, alongside structural tags for constrained generation?

Thanks for the great discussion here! Previously xgrammar had a proposal for parsing some output text with the structural tag at mlc-ai/xgrammar#303. The API is like

def parse_with_structural_tag(input_str, structural_tag) -> StructuralTagResult: pass

It's not finished yet, but it should be easy to pick it up recently.

This maps the output text to the structure of the StructuralTag. To further use it in tool calling (and also parallel tool calling), we still need to map it into the OpenAI format (the "tool_calls": format you mentioned).

I agree with you that with regex/ebnf the output is harder to parse. It's possible with xgrammar's builtin Earley parser, but may introduce multiple possible parsing result and we need to determine which one to use. Maybe we can restrict the parser with StructuralTags without any regex/ebnf content.

One possible and suitable solution is, we can have an extra layer of abstraction for tool calling and further lower that into structural tag, and use the structural tag parser to handle that, then convert it to OpenAI tool calling format. The benefit could be: 1) it's easier to handle the different tool calling formats of different models; 2) we can unify the constraint decoding (or guided decoding) and tool calling parsing.

Quick update here: I experimented with dropping this PR and instead adding parsing support to XGrammar constrained generation schemas. However, after experimentation, I think there's a big difference between the schemas needed for constrained generation and the schemas needed for output parsing. In particular, constrained generation schemas need a lot of knowledge about tool argument names and types if they want to ensure that all tool calls are valid, but a parser can skip all this and simply record the model output with a generic schema that works for all tools.

I think constrained generation schemas are still very useful! But I'm much less confident that it's a good idea to overload them with both tasks; there's actually a lot of friction between them. It probably makes more sense for models to have both a "response schema" and a "generation schema", and this PR will remain focused on the first.

Going to merge this one after addressing most of the reviewer comments! I think the docs could still use a little more cleanup, but the core is solid, and I'd like this to get into a release sooner rather than later so I can start merging PRs to add response schemas to models. Since only a few models will have response schemas to start and I'll likely be writing all of them, we can treat the feature as experimental/unstable for now, and if I need to change anything in the spec I can go back and change the schemas too.